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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  ot  publication  am  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  .are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  Fort  Knox  Field  Unit  has  a  broad  concern  for  the  effectiveness  and 
efficiency  of  both  training  programs  in  institutional  and  field  settings  and 
training  guidance  materials  for  initial  and  sustainment  training.  One  specific 
concern  is  with  the  adequacy  of  the  information  exchange  between  operating  units 
in  the  field  and  Centers/Schools. 

The  feedback  to  Army  Centers/Schools  is  currently  somewhat  disorganized  and 
largely  based  upon  subjective  data.  There  is  a  need  for  an  integration  of  methods 
since  no  method  is  sufficient  by  itself  and  no  existing  records  provide  useful 
task-specific  feedback  data. 

This  report  is  a  review  of  current  practices  regarding  the  feedback  of 
soldier  performance  data  from  operating  units  to  Centers/Schools.  Research  was 
conducted  to  define  these  information  requirements  and  to  develop  procedures  for 
generating  the  necessary  data,  feeding  it  back  from  operating  units  to  Centers/ 
Schools,  managing  the  resulting  data  base,  and  drawing  implications  from  the  data 
base  to  aid  in  making  decisions  regarding  training  program  or  training  guidance 
modifications. 

The  results  of  this  program  will  be  useful  to  training  developers  and 
evaluators  for  implementing  an  integrated  task-specific  feedback  system  based, 
as  much  as  possible,  upon  objective  data.  Data  from  such  a  system  will  be  used 
to  revise  training  programs  and  materials  to  meet  the  needs  of  the  field. 
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FIELD  PERFORMANCE  FEEDBACK  -  A  PROBLEM  REVIEW 

BRIEF 


Requirement: 


In  order  to  assess  the  quality  of  their  products,  including  both  graduates 
r.n d  training  materials.  Army  Ceaters/Schools  need  meaningful  feedback  from 
field  units.  In  this  report  six  methods  of  gathering  field  performance  feed¬ 
back  are  reviewed,  the  general  aspects  of  an  integrated  feedback  system  are 
discussed,  and  recommendations  for  future  research  are  developed. 

? r :  -  ^Lre  and  Findings: 


six  feedback  methods  are  discussed  using  data  from  three  sources: 

...  .  of  the  relevant  literature,  results  of  previous  research,  and  struc- 

. :  .  d  interviews  of  seven  battalion  commanders.  The  general  findings  are  that 
feeloeck  to  Army  Centers/Schools  is  currently  somewhat  disorganized  and  largely 
based  upon  subjective  data,  there  is  need  for  an  integration  of  methods  since 
no  method  is  sufficient  by  itself,  and  no  existing  records  provide  useful  task- 
specific  feedback  data,  except  perhaps  SQT  results.  Future  research  directions 
identified  include  the  integration  of  feedback  data  collection  with  unit 
activities,  such  as  crew  and  platoon  drills,  an  assessment  of  the  accuracy  of 
subjective  feedback  data,  and  the  development  of  methods  for  managing  and 
utilizing  feedback  information. 

Utilization  of  Findings: 

The  findings  of  this  report  will  be  used  to  guide  a  field  performance 
feedback  research  program  in  the  directions  outlined  above.  The  results  of 
this  program  will  be  useful  to  training  developers  and  evaluators  for  imple¬ 
menting  an  integrated  task-specific  feedback  system  based  as  much  as  possible 
upon  objective  data.  Data  from  such  a  system  will  be  used  to  revise  training 
programs  and  materials  to  meet  the  needs  of  the  field. 
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FIELD  PERFORMANCE  FEEDBACK  -  A  PROBLEM  REVIEW 


INTRODUCTION 

In  order  to  assess  the  effectiveness  of  their  training,  programs,  and 
materials,  Army  schools  need  meaningful  feedback  from  the  field  on  the  quality 
of  their  products.  These  products  include  both  graduates  and  training  doc¬ 
trine,  guidance,  or  materials.  The  military  services  have  perceived  the  need 
for  feedback  from  the  field  and  have  approached  gathering  It  in  various  ways, 
ranging  from  receipt  of  informal  comments  from  field  commanders  to  development 
of  formal  reporting  systems  (see  Hall,  Lam,  and  Bellomy  (1976)  for  an  overview 
of  approaches  in  the  various  services).  A  variety  of  approaches  have  also 
been  utilized  within  Army  schools,  and  they  have  typically  been  applied  by 
Directorates  of  Evaluation  or,  in  some  cases,  by  specific  offices  formed  to 
gather  training  effectiveness  data  from  institutional  and  field  units  (e.g., 
the  Office  of  Armor  Force  Management  and  Standardization  (OAFKS)  at  Fort  Knox). 
The  fact  that  a  variety  of  feedback  approaches  are  in  use  suggests  that  none  of 
them  has  been  completely  satisfactory,  especially  in  the  light  of  the  numerous 
training  problems  which  still  exist  in  the  Army.  The  purpose  of  the  present 
paper  is  to  review  the  various  feedback  approaches  which  have  been  or  could  be 
utilized  in  the  Army,  to  assess  their  strengths  and  weaknesses,  and  to  propose 
integrated  approaches  which  can  optimize  the  utility  of  field  performance  feed¬ 
back.  Put  more  simply,  the  purpose  is  to  review  the  status  of  feedback  systems 
and  suggest  ways  to  improve  them.  The  feedback  systems  addressed  will  pri¬ 
marily  be  external  ones,  involving  the  flow  of  information  from  field  units  to 
Army  Centers/Schools.  Of  course,  the  flow  of  information  from  Centers/Schools 
to  field  units  must  also  be  addressed  to  some  extent,  since  feedback  is  a  two- 
way  street.  The  flow  of  internal  feedback  within  a  Center/ School  will  not  be 
a  focus  of  the  present  paper. 

Several  characteristics  of  an  ideal  field  performance  feedback  system  can 
be  identified.  The  data  gathered  by  such  a  system  should  be  as  objective  as 
possible  and  should  have  high  validity,  so  that  they  represent  reality.  The 
data  should  also  be  task-specific  rather  than  general  in  nature,  so  that 
specific  training  needs  can  be  identified.  The  data  collection  and  analysis 
approach  should  operate  quickly  enough  to  provide  a  true  picture  of  current 
field  performance.  Personnel  turnover  and  skill  retention  loss  rates  are  so 
nigh  in  today's  Army  that  the  performance  o"f  field  units  can  change  rapidly. 

The  collection  of  field  performance  feedback  data  should  not  be  a  detractor  to 
training,  but  should  be  integrated  into  the  normal  training  cycle  as  much  as 
possible.  A  truly  integrated  feedback  system  should  apply  to  all  levels  of 
Army  training,  from  initial-entry  training  to  the  US  Army  War  College,  and 
should  also  apply  to  all  specialties,  both  officer  and  enlisted.  The  feedback 
approaches  developed  must  be  doable  within  available  resources,  or  specified  in 
a  stepwise  fashion  that  is  tailorable  to  the  resources  available.  But  the 
approaches  must  allow  collection  of  data  from  a  large  enough  sample  of  units 
to  determine  overall  trends  and  problems  which  are  not  unique  to  specific 
units.  The  feedback  system  should  also  include  ways  to  ensure  that  it  has  an 
impact;  i.e.,  it  should  include  criteria  for  making  decisions  based  upon  the 
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data  gathered  and  methods  for  implementing  them.  Approaches  for  collating 
data  to  reach  proper  decisions  and  for  integrating  collective  and  individual 
task  data  should  also  be  specified.  From  another  perspective,  a  field  perform¬ 
ance  feedback  system  should  specify  what  data  are  to  be  collected,  where  and 
when  they  should  be  collected,  procedures  for  collating  data,  appropriate  de¬ 
cision  criteria,  and  methods  for  communicating  decisions  to  appropriate  action 
agencies.  All  these  parameters  will  be  addressed  in  the  present  paper,  although 
complete  specification  of  them  will  obviously  have  to  await  the  results  of 
further  research. 

Approaches  to  field  performance  feedback  will  be  discussed  in  the  present 
paper  in  the  light  of  a  review  of  the  literature,  interactions  with  OAFMS  per¬ 
sonnel,  and  data  gathered  from  three  sources:  the  Army  Training  Study  (ARTS), 
the  application  of  living  systems  theory  (LST)  to  analysis  of  battalions*  pro¬ 
cesses,  and  interviews  of  battalion  commanders  and  staffs.  One  facet  of  the 
ARTS  (7.978)  involved  the  administration  of  written  and  hands-on  performance 
tests  to  personnel  in  Armor  units  in  order  to  assess  their  retention  of  in¬ 
stitutional  training.  These  performance  data  and  related  survey  and  experience 
data  gathered  in  the  same  project  are  relevant  to  particular  aspects  of  field 
performance  feedback.  Also,  during  the  past  two  years  the  US  Army  Training 
and  Doctrine  Command  (TRADOC)  has  sponsored  a  program  of  research  to  assess 
the  utility  of  LST  for  evaluating  the  quality  of  information  and  materiel/ 
resources  processing  in  various  types  of  battalions  (especially  Armor).  The 
specifics  of  the  application  of  LST  are  not  of  interest  here  (see  Ruscoe, 

Giguet,  Brown,  Burnside,  and  Cary,  1979),  but  the  use  of  a  general  systems 
theory  led  to  collection  of  a  large  amount  of  data  which  are  relevant  here. 
Almost  all  the  major  types  of  data  traditionally  maintained  at  battalion  level 
were  gathered  during  this  research,  as  well  as  various  types  of  survey  and 
interview  data.  As  a  further  source  of  data  for  the  present  paper,  structured 
interviews  were  conducted  with  the  commanders  and/or  S3's  (operations  officers) 
of  seven  battalions  of  various  types  (infantry,  armor,  artillery,  maintenance, 
support,  and  engineer)  at  Fort  Knox,  KY  during  May  1981.  These  interviews  ad¬ 
dressed  areas  such  as  feedback  between  battalions  and  the  appropriate  Centers/ 
Schools  on  the  proficiency  of  graduates  and  the  usefulness  of  training  guidance/ 
materials,  the  availability  and  utility  of  various  types  of  existing  records 
for  feedback  purposes.,  and  other  information  relating  to  individual  soldier 
proficiency  at  the  task  level.  Through  these  interviews  an  update  was  obtained 
on  the  current  status  of  feedback  between  battalions  and  Centers/Schools,  and 
the  information  base  was  extended  beyond  Armor  units. 

In  the  remaining  sections  of  this  paper  approaches  to  obtaining  field 
performance  feedback  will  be  categorized  and  discussed,  using  data  gathered  in 
the  research  projects  and  structured  interviews  briefly  described  above  as 
examples.  The  desired  characteristics  of  a  feedback  system  will  then  be  fur¬ 
ther  discussed,  followed  by  development  of  initial  research  proposals  for  ob¬ 
taining  task-specific  feedback  from  a  large  enough  sample  of  field  units  to 
allow  discernment  of  meaningful  trends  and  patterns.  The  conclusions  developed 
are  preliminary  ones  based  upon  a  total  sample  of  21  battalions,  most  of  which 
were  Armor  units.  But  the  approaches  suggested  should  provide  a  logical  start 
toward  improvement  of  feedback  between  field  units  and  Army  Centers/Schools. 
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FEEDBACK  METHODS 


A  review  of  the  literature  indicates  that  there  are  at  least  six  methods 
for  gathering  field  performance  feedback.  These  are:  (1)  the  receipt  of  un¬ 
solicited  informal  comments  from  personnel  in  field  units;  (2)  the  analysis  of 
existing  records  traditionally  maintained  by  field  units;  (3)  the  administra¬ 
tion  of  surveys  or  questionnaires  to  field  personnel;  (4)  the  conduct  of 
structured  or  unstructured  interviews  with  field  personnel,  either  at  their 
field  sites  or  while  they  are  attending  courses  at  the  Center/School;  (5)  the 
observation  and  recording  of  performance  in  the  field;  and,  (6)  the  testing  of 
graduates  in  the  field  using  paper-and-pencil  and/or  hands-on  tests.  Each  of 
these  methods  is  discussed  in  detail  below,  using  examples  from  data  sources 
described  above.  Other  methods  could  be  suggested,  but  they  are  not  considered 
feasible  for  present  purposes.  For  example.  Dyer,- Ryan,  and  Mew  (1975)  have 
suggested  that  graduates  could  provide  feedback  to  Centers /Schools  by  maintain¬ 
ing  diaries  of  their  field  performance  successes  and  failures.  This  approach 
is  not  feasible,  given  the  limited  writing  abilities  and  time  of  many  of  today*s 
soldiers.  However,  a  related  approach  that  may  be  feasible,  the  maintenance 
of  job  books,  will  be  discussed  below. 

Unsolicited  Informal  Comments.  A  good  deal  of  information  flow  in  the 
military  travels  through  informal  channels.  Military  personnel  communicate 
information  and  comments  to  acquaintances  at  Centers/Schools  or  in  other  staff 
positions  through  telephone  calls  and  chance  face-to-face  meetings  while  in 
travel  status.  This  is  sometimes  referred  to  as  the  "old  boy"  network.  Use 
of  such  a  network  was  addressed  in  the  interviews  of  seven  battalion  commanders 
described  earlier;  they  were  asked  to  describe  any  unsolicited  informal  comments 
they  may  have  communicated  to  a  Center/School  in  the  previous  year  regarding 
ube  proficiency  of  its  graduates  or  the  usefulness  of  its  training  materials. 

The  reported  amount  of  such  interaction  varied  from  almost  none  to  almost  con¬ 
tinuous,  depending  largely  upon  the  attitude  of  the  battalion  commander.  Some 
commanders  have  many  informal  contacts  at  the  Center/School  and  feel  free  to 
communicate  with  them  frequently,  while  others  feel  that  they  should  work  with 
what  they  get  and  not  give  the  appearance  of  complaining.  Five  of  seven  bat¬ 
talion  commanders  reported  that  they  had  sent  unsolicited  comments  to  a 
Center/School  on  the  proficiency  of  its  graduates,  and  four  of  seven  had  sent 
comments  on  the  usefulness  of  training  materials.  On  the  average,  commanders 
reported  transmitting  three  or  four  comments  per  year  in  each  area,  usually 
through  phone  calls  or  meetings  at  conferences.  The  comments  dealt  with  such 
areas  as  suggested  revisions  in  tasks  to  be  trained  at  Centers/ Schools,  changes 
or  clarifications  of  training  doctrine,  revisions  of  tactics,  and  suggested 
changes  in  use  of  new  equipment.  Those  commanders  who  frequently  sent  informal 
comments  felt  that  the  Center/School  was  usually  responsive  to  their  suggestions. 

The  receipt  of  unsolicited  informal  comments  is  obviously  not  an  adequate 
method  for  obtaining  field  performance  feedback.  This  approach  depends  too 
much  upon  the  attitude  of  the  field  commander  and  the  number  of  informal  con¬ 
tacts  he  has  established  at  the  appropriate  Center/School.  Information 
gathered  in  this  approach  may  over-represent  particular  units  and  under¬ 
represent  'others,  thus  failing  to  provide  an  accurate  picture  of  the  overall 
situation  in  the  field.  The  data  provided  may  be  sporadic,  not  objective. 
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general  rather  than  task-specific ,  and  difficult  to  pull  together.  On  the 
other  hand,  this  approach  has  the  advantages  of  requiring  few  resources  and 
not  detracting  from  training.  Informal  communications  between  field  units  and 
Centers/Schools  are  going  to  occur,  and  they  should  be  encouraged  and  utilized 
to  best  advantage,  as  long  as  they  are  by  no  means  considered  an  adequate  or 
complete  basis  for  field  performance  feedback  in  and  of  themselves. . 

One  way  in  which  informal  feedback  can  be  encouraged  is  through  the  medium 
of  regular  Command  Letters.  The  Commanding  Generals  of  the  Infantry,  Armor, 
and  other  Centers/ Schools  send  information  letters  to  field  commanders  on  a 
regular  basis.  Feedback  on  current  issues  is  sometimes  encouraged  or  requested 
in  these  letters,  and  this  approach  could  be  expanded  in  the  future  to  request 
feedback  on  more  specific  topics.  Another  important  informal  feedback  medium 
is  attendance  at  conferences  given  by  Centers/Schools;  commanders  interviewed 
who  had  attended  such  conferences  indicated  that  the  meetings  provided  oppor¬ 
tunities  for  communicating  feedback  and  establishing  personal  contacts* 
Participation  in  workshops  and  informal  visits  to  Centers/Schools  were  also 
reported  as  very  useful  by  battalion  commanders  interviewed.  Field  commanders 
should  be  encouraged  to  visit  the  appropriate  Center/School  as  time  and  re¬ 
sources  allow,  especially  in  light  of  the  fact  that  the  length  of  command  tours 
is  increasing.  Another  way  in  which  informal  feedback  can  be  encouraged  is 
through  the  use  of  "hot  lines,"  which  have  been  established  by  many  Centers/ 
Schools.  Such  direct  lines  should  be  widely  publicized  and  their  use  strongly 
encouraged.  A  final  way  in  which  the  occurrence  of  informal  feedback  might  be 
increased  would  be  the  development  of  specific  formats  for  it.  One  must  move 
cautiously  in  this  direction,  since  the  specification  of  rigid  formats  would 
destroy  the  very  nature  of  informal  feedback.  But  some  degree  of  formalization 
could  be  accomplished.  For  example,  the  US  Air  Force  has  utilized  a  Training 
Quality  Report  on  which  field  personnel  can  report  the  overtraining  or  under¬ 
training  of  newly  arriving  personnel  directly  to  the  Air  Training  Command  (the 
report  is  described  by  Hall,  Lam,  and  Bellomy,  1976).  The  availability  of  such 
a  report  format  in  the  Army  might  increase  the  amount,  -speed,  and  specificity 
of  informal  feedback.  In  summary,  use  of  methods  such  as  those  described  above 
might  optimize  the  small  but  Important  role  of  unsolicited  informal  comments 
in  a  field  performance  feedback  system. 

Analysis  of  Existing  Records.  One  could  argue  that  there  is  no  need  to 
generate  new  data  for  a  field  performance  feedback  system,  since  sufficient 
data  are  already  collected  and  maintained  at  battalion  level.  These  data  and 
records  will  be  reviewed  in  this  section  in  order  to  show  that  this  argument 
is  generally  not  a  sound  one.  The  types  of  existing  records  available  at  bat¬ 
talion  level  include  Army  Training  and  Evaluation  Program  (ARTEP)  results, 
results  of  inspections,  qualification  results,  data  recorded  in  job  books. 

Skill  Qualification  Test  (SQT)  results,  training  and  maintenance  records,  and 
data  recorded  on  Unit  Status  Reports  (USR's).  Each  of  these  types  of  records 
will  be  discussed  below  using  information  gathered  during  LST  research  and 
interviews  with  battalion  commanders.  During  a  portion  of  the  LST  research, 
most  of  the  data  traditionally  maintained  at  battalion  level  was  gathered  from 
14  Armor  battalions.  During  Interviews  with  battalion  commanders,  the  format, 
availability,  and  degrea  of  specificity  of  these  types  of  records  wore 
addressed. 
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ARTEP  results  provide  an  indication  of  a  unit's  effectiveness  or  readiness, 
but  experience  has  shown  that  they  are  not  always  available  and  are  not  de¬ 
finitive  enough  to  provide  adequate- field  performance  feedback.  In  collecting 
data  from  Armor  units  during  LST  research,  it  was  found  that  ARTEP  results  were 
not  available  from  over  half  the  battalions  visited.  Two  of  seven  battalion 
commanders  interviewed  at  Fort  Knox  reported  that  their  battalion  had  never 
participated  in  an  ARTEP.  In  some  cases  a  nonstandard  ARTEP  had  been  conducted 
during  the  past  year,  and  in  others  detailed  results  had  not  been  provided  to 
the  battalion  commander  (this  was  done  so  that  the  results  would  not  be  used  as 
evaluations  of  company/battery  commanders).  Regardless  of  the  reasons,  the 
general  lesson  learned  is  that  ARTEP  results  are  not  always  available  at  bat¬ 
talion  level.  When  ARTEP  results  are  available,  they  are  usually  in  a  general 
format  listing  strengths  and  weaknesses  for  the  major  elements  of  a  battalion. 
Various  factors  may  not  be  observed  or  evaluated  consistently  for  all  bat¬ 
talions  or  elements  thereof.  For  example,  during  LST  research  it  was  found 
that  as  many  as  50%  of  the  standard  evaluation  factors  were  not  observed  in 
some  companies.  It  is  difficult  to  oevelop  a  quantitative  objective  measure 
from  such  results  to  allow  summaries  across  battalions.  The  strengths  and 
weaknesses  listed  are  often  very  general  (e.g.,  "staff  was  not  fully  integrated 
in  the  planning  process"  or  "supplemental  missions  were  generally  weak")  and 
do  not  provide  any  task-specific  feedback  on  how  to  correct  problems.  Ratings 
are  usually  provided  for  each  company  on  performance  of  major  missions  (e.g., 
"movement  to  contact"  or  "defense"),  but  there  is  no  listing  of  specific  in¬ 
dividual  or  crew-level  tasks  which  need  further  training.  Only  one  of  seven 
battalion  commanders  interviewed  indicated  that  individual  task-specific  per¬ 
formance  feedback  could  be  derived  from  ARTEP  results.  The  general  perception 
was  that  the  relationships  of  ARTEP  and  individual  tasks  are  becoming  clearer 
in  new  ARTEP  manuals,  but  are  still  not  completely  specified.  A  final  area  in 
which  the  utility  of  ARTEP  results  as  feedback  can  be  questioned  relates  to 
their  objectivity.  ARTEP  results  are  based  upon  observations  made  by  personnel 
from  a  unit  similar  to  the  one  being  evaluated,  and  such  observations  are  not 
as  detailed  or  objective  as  needed  for  purposes  of  training  feedback.  Since 
ARTEP  results  are  not  consistently  available,  are  not  recorded  in  task-specific 
detail,  are  not  completely  objective,  and  are  difficult  to  summarize  across 
battalions,  the  appropriate  conclusion  is  that  they  have  limited  utility  as 
feedback  to  Centers/Schools.  ARTEP' s  are  important  training  exercises,  but  as 
presently  conducted  they  should  not  be  viewed  as  useful  elements  of  field  per¬ 
formance  feedback.  Their  feedback  utility  may  be  increased  in  the  future  by 
making  the  evaluations  more  consistent  and  objective  (i.e.,  by  standardizing 
recording,  reporting,  and  analysis  of  ARTEP  results)  and  by  further  specifying 
the  relationships'  of  ARTEP  and  individual  tasks. 

With  regard  to  results  of  inspections,  the  primary  type  available  in 
field  units  is  results  of  the  Annual  General  Inspection  (AGI).  The  AGI  pri¬ 
marily  addresses  administrative  and  managerial  procedures,  thus  providing  a 
general  evaluation  of  the  operations  of  a  battalion  and  its  major  elements. 

But  little  or  r.o  task-specific  evaluation  of  training  is  provided.  Problems 
identified  in  AGI  results  are  typically  general  in  nature  (e.g.,  "management 
procedures  were  inadequate"  or  "preventive  maintenance  services  were  not  being 
performed  in  a  timely  manner"),  and  it  is  difficult  to  relate  them  to  specific 
revisions  needed  in  institutional  training.  It  is  also  difficult  to  reduce 
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AGI  results  to  a  quantitative  measure  which  can  be  summarized  across  units. 
During  LSI  research,  the  measure  derived  was  the  percentage  of  rated  areas 
which  were  given  a  satisfactory  rating.  This  is  not  an  adequate  measure, 
since  the  number  and  type  of  areas  rated  vary  greatly  from  unit  to  unit.  The 
number  of  areas  rated  varied  from  14  to  99  in  battalions  included  in  LST  re¬ 
search,  and  the  types  of  areas  rated  included  repair  parts,  tactical  support 
vehicles,  sanitation,  safety,  crime  prevention,  personnel  data  cards,  and 
billets.  It  is  difficult  to  develop  meaningful  training  feedback  from  such  a 
variety  of  evaluations.  The  battalion  commanders  interviewed  agreed  with  .this 
conclusion,  since  all  seven  indicated  that  little  or  no  task-specific  training 
feedback  could  be  derived  from  AGI  results.  Another  type  of  inspection  men¬ 
tioned  frequently  by  these  commanders  was  Logistical  Assistance  and  Assessment 
Team  (LAAT)  and  Training  Assistance  and  Assessment  Team  (TAAT)  visits,  but 
again  the  general  perception  was  that  results  would  have  little  utility  as 
feedback  to  Centers/Schools.  Commanders  were  also  asked  whether  their  unit 
conducts  training  inspections,  and  six  of  seven  indicated  that  they  do.  These 
inspections  are  conducted  by  the  battalion  commander,  S3,  company  commanders, 
or  senior  NCO's,  and  it  was  estimated  that  an  average  of  20%  of  unit  training 
is  inspected.  A  general,  one-page  format  is  used  for  this,  and  records  are 
not  maintained  permanently,  so  these  results  are  also  not  very  useful  as  field 
performance  feedback.  In  conclusion,  results  of  inspections  are  generally  not 
appropriate  or  useful  for  feedback  purposes,  and  they  will  not  ue  in  the  future 
unless  they  are  made  more  objective,  standardized,  and  task-specific. 

There  are  various  levels  of  qualification  results  which  could  be  con¬ 
sidered  as  candidates  for  training  feedback.  In  Armor  units  Table  VIII  (crew 
qualification  on  firing  of  main  gun)  results  provide  an  indicator  of  crew  per¬ 
formance  and  thus  a  measure  of  training  effectiveness  at  this  level.  Table  IX 
(platoon  qualification  on  firing  and  movement)  results  provide  a  similar  in¬ 
dicator  for  platoon  performance.  Based  upon  data  gathered  in  previous  research, 
these  results  are  not  as  useful  for  training  feedback  purposes  as  one  would 
hope,  since  information  on  individual  task  performance  is  not  maintained. 

Scoring  problems  and  subjective  evaluations  inherent  in  such  qualification 
exercises  permit,  at  best,  only  collective  task  performance  estimates.  Targets 
are  often  obscured  by  dust  ?nd  smoke  and  the  determination  of  hits  and  misses 
is  sometimes  difficult  (see  Eaton  and  Whalen,  1980).  Control  of  fire  and 
movement  techniques  are  examples  of  areas  which  are  to  some  extent  evaluated 
subjectively.  Another  possible  problem  is  differences  in  the  way  gunnery 
qualifications  are  conducted  in  different  locations,  due  to  limits  on  range 
availability  and  other  local  constraints  (Kalergis,  1977).  A  data  collector 
cannot  through  the  mail  or  in  person  simply  request  gunnery  qualification  re¬ 
sults  and  expect  to  get  comparable  data  across  units.  He  or  she  must  also 
collect  information  on  how  gunnery  qualification  was  conducted  and  scored. 

During  LST  research,  a  broad  request  for  gunnery  qualification  data  resulted 
in  provision  of  results  from  various  types  of  Table  VIII,  Table  IX,  and  un¬ 
determined  exercises.  In  one  iteration  of  this  research,  three  of  eight  bat¬ 
talions  provided  gunnery  qualification  data  that  were  questionable  and  were 
apparently  not  from  Table  VIII  (they  were  from  Table  IX  or  an  experimental 
Table) .  These  data  were  not  usable  due  to  a  lack  of  comparability  across 
units.  Another  problem  with  gunnery  qualification  results  alluded  to  earlier 
is  that  they  are  generally  not  task-specific  or  definitive  enough  to  allow 
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identification  of  specific  training  problems  or  needs.  Such  results  are  g-&cx  - 
ally  maintained  in  terms  of  number  or  percentage  of  crews  or  platoons  qu. 
or  point  totals  for  each  crew  or  platoon.  Specific  problems  or  reasons  ior 
failure  to  qualify  can  only  be  determined  by  examining  original  scorecards  . 
interviewing  unit  personnel.  Additional  task-specific  feedback  might  be  ob¬ 
tained  from  scorecards  for  gunnery  exercises  preliminary  to  Tables  VIII  ar.d  .'X, 
but  such  data  are  not  standardly  maintained  across  units.  The  utility  of  gun¬ 
nery  results  as  training  feedback  could  be  increased  by  collecting  and  main¬ 
taining  them  in  task-specific  detail  throughout  and  following  the  entire  gun¬ 
nery  cycle.  Previous  experience  has  indicated  that  detailed  results  are  dis¬ 
carded  shortly  after  overall  qualification  scores  are  determined.  Simple 
standardized  data  collection  and  feedback  forms  could  easily  be  designed  Co 
increase  the  availability  of  such  data.  In  summary,  gunnery  results  a re  not 
generally  adequate  for  training  feedback  purposes  because  of  inter-unit  dif¬ 
ferences  in  conducting  gunnery  tables  and  because  task-specific  results  are 
either  not  available  or  require  considerable  effort  to  obtain.  Two  Armor  bat¬ 
talion  commanders  interviewed  at  Fort  Knox  agreed  with  this  conclusion  by 
stating  that  only  limited  individual  task  performance  data  could  be  derived 
from  gunnery  results.  Other  commanders  interviewed  indicated  that  there  are 
presently  no  collective  qualification  results  available  in  other  types  of  units 
which  provide  field  performance  feedback  at  the  individual  task  level. 

Various  individual  qualification  results  are  gathered  periodically  in 
field  units,  and  the  results  are  more  task-specific  in  nature  than  are  the 
gunnery  data  discussed  above.  Examples  of  these  sorts  of  data  are  individual 
weapons  qualifications,  results  of  physical  fitness  tests,  and  records  of  man¬ 
datory  training.  Such  results  are  important  for  the  maintenance  of  unit  readi¬ 
ness,  and  previous  data  collection  experience  and  interviews  of  commanders  in¬ 
dicate  that  they  are  generally  readily  available.  But  they  are  not  antici¬ 
pated  to  be  particularly  useful  elements  of  field  performance  feedback  for  a- 
variety  of  reasons.  The  tasks  included  in  such  qualification  data  are  mostly 
basic  ones  which  are  not  of  primary  interest  to  specific  Centers/Schools. 
Qualification  test  conduct  and  data  maintenance  may  not  be  standard  from  unit 
to  unit.  In  many  cases,  an  overall  test  result  may  be  recorded  without  task- 
specific  detail.  So  individual  qualification  test  results  are  readily  avail¬ 
able,  but  they  should  be  considered  as  only  a  minor  element  of  a  performance 
feedback  system. 

Another  type  of  existing  records  which  might  appear  to  be  useful  for 
feedback  purposes  is  recordings  in  job  books.  Supervisors  observe  and  infor¬ 
mally  test  their  personnel  and  record  "GO's,"  "NO  GO's,"  dates,  and  comments 
in  each  individual’s  job  book.  These  data  are  recorded  for  specific  tasks, 
and  training  needs  or  problems  are  frequently  addressed  in  comments.  Job  book 
data  were  considered  but  not  collected  during  LST  research,  primarily  because 
of  the  extensive  resources  required  to  do  so.  Interviews  of  battalion  com¬ 
manders  indicated  that  job  book  recordings  are  generally  not  consolidated  at 
battalion  level;  only  one  of  seven  battalions  had  made  any  effort  in  this 
direction.  Collecting  job  book  data  throughout  a  battalion  and  consolidating 
them  by  task  would  thus  require  considerable  tia  and  effort,  A  sample  of  such 
data  could  be  gathered  within  a  battalion,  but  great  care  should  be  exercised 
in  assuring  the  sample  to  be  a  representative  one.  Another  problem  with  job 


book  data  is  the  fact  that  they  may  not  be  completely  accurate  or  reliable. 
During  the  turmoil  of  daily  activities,  supervisors  do  not  have  time  to  make  a 
job  book  entry  every  time  they  observe  one  of  their  personnel  performing  or 
failing  a  task.  Entries  are  typically  made  from  memory  at  the  end  of  the  day 
or  week.  It  is  possible  that  some  entries  are  never  made  until  required  by  an 
external  agent,  and  job  books  may  not  be  kept  up-to-date  in  many  units.  If 
job  book  data  were  to  be  gathered  on  a  certain  date,  many  entries  might  hur¬ 
riedly  be  made  the  night  before,  resulting  in  inaccuracies.  The  seven  bat¬ 
talion  commanders  interviewed  supported  these  concerns  by  reporting  that  the 
accuracy  and  completeness  of  job  book  entries  are  highly  variable,  depending 
largely  upon  the  attitudes  and  writing  abilities  of  supervisors.  Commanders 
in  general  questioned  the  usefulness  of  job  books;  one  stated  that  they  would 
be  useful  only  if  individual  training  could  be  conducted  on  a  daily  basis, 
since  most  skills  undergo  rapid  retention  loss.  The  appropriate  conclusion  is 
thus  that  job  book  data  are  not  available  or  accurate  enough  to  be  a  useful 
element  of  field  performance  feedback. 

SQT  results  are  one  type  of  existing  record  which  can  provide  useful  task- 
specific  feedback.  In  the  current  system,  individuals  are  provided  feedback 
on  their  results  and  battalions  are  provided  unit  summaries  by  task.  All  bat¬ 
talion  commanders  interviewed  indicated  that  they  receive  such  reports,  and 
most  reported  that  they  use  them  to  manage  training.  For  example,  a  task  show¬ 
ing  a  high  failure  rate  on  the  SQT  may  be  emphasized  in  unit  training  for  a 
few  weeks.  Some  commanders  indicated  that  the  turnaround  time  for  receiving 
CQT  results  is  a  problem;  if  it  takes  60  days  to  receive  feedback  on  results 
(as  it  reportedly  sometimes  does),  the  currently  high  personnel  turnover  and 
forgetting  rates  may  make  the  results  unrepresentative  of  the  current  training 
status  of  unit  personnel.  Turnaround  time  is  apparently  improving,  though. 
Commanders  also  expressed  a  need  for  feedback  on  SQT  results  across  broader 
levels  (e.g.,  brigade),  in  order  to  provide  a  baseline  to  compare  their  units 
against.  Discussions  with  SQT  managers  have  indicated  that  quarterly  reports 
on  SQT  results  are  provided  at  brigade,  Center/School,  and  major  command 
(MACOM)  levels.  The  extent  to  which  such  reports  are  currently  used  to  revise 
training  is  not  known,  but  this  is  an  important  element  of  a  field  performance 
feedback  system,  since  results  are  provided  by  task.  One  problem  with  SQT 
data  that  was  noted  in  interviews  and  previous  research  is  that  results  are 
not  yet  available  for  all  MOS's.  During  LST  research  in  Armor  battalions,  SQT 
results  were  available  for  various  low-density  KOS’s,  but  were  not  always 
available  for  high-density  MOS’s,  such  as  the  19  series.  In  eight  Armor  bat¬ 
talions  visited  during  the  summer  of  1980,  three  had  conducted  no  SQT's  for 
Armor  MOS's  during  the  past  year,  and  an  average  of  only  about  70  personnel 
having  an  Armor  MOS  had  completed  an  SQT  in  each  battalion  during  that  time¬ 
frame.  Also,  approximately  40%  of  the  SQT  data  available  represented  low- 
density  MOS’s.  SQT's  are  still  being  developed  and  revised  for  some  MOS's,  so 
stable  long-term  results  are  not  yet  available.  In  addition,  it  is  likely  that 
the  reliability  of  SQT  results  has  not  been  adequately  addressed,  at  least  for 
some  MOS's  (see  Nieva,  Myers,  and  Glicknan  (1979)  for  discussion  of  scoring 
and  other  problems).  But,  under  the  assumption  that  the  availability  and  re¬ 
liability  of  SQT  results  will  be  increased,  these  data  should  be  included  in 
any  feedback  system  developed,  especially  since  a  method  already  exists  for 
providing  them  to  Centers/Schools.  Training  programs  could  be  revised  to 


increase  the  amount  of  training  or  improve  training  materials  for  tasks  show¬ 
ing  high  SQT  failure  rates.  Future  research  should  address  how  SQT  results 
can  be  integrated  withiu  field  performance  feedback  system  and  how  they  can 
best  be  utilized. 

During  structured  interviews  battalion  commanders  were  asked  whether  any 
other  unit  training  records,  in  addition  to  those  discussed  above,  would  be 
useful  as  feedback  to  Cen.ters/Schools,  The  general  response  was  that  unit 
training  records  are  minimized  in  the  current  environment,  and  are  thus  not 
available  for  feedback.  Records  are  kept  of  results  of  exercises  such  as  mini- 
SQT’s  and  platoon-level  ARTEP's  in  some  cases,  but  they  are  not  consistently 
maintained  and  are  highly  perishable.  The  interview  responses  and  previous 
data  collection  experience  during  ARTS  and  LST  research  support  the  conclusion 
that  no  other  unit  training  records  are  useful  for  feedback  purposes.  Battalion 
commanders  were  also  asked  about  the  utility  of  maintenance  records  for  feed¬ 
back.  They  generally  responded  that  while  records  such  as  vehicle  logbooks 
are  maintained,  they  tend  to  be  fragmented,  not  permanent,  and  difficult  to 
interpret.  That  is,  if  a  particular  maintenance  problem  continues  to  crop  up, 
it  is  difficult  to  attribute  it  to  a  problem  with  equipment  or  a  problem  in 
the  training  of  mechanics.  It  is  therefore  concluded  that  maintenance  records 
will  not  provide  a  useful  element  of  a  field  performance  feedback  system. 

Several  commanders  interviewed  stated  that  interviews  with  maintenance  super¬ 
visors  would  provide  the  most  useful  information  in  this  area.  Interview 
methodology  is  discussed  in  a  later  section  of  this  paper. 

Another  type  of  existing  record  which  should  be  briefly  considered  here 
is  the  Unit  Status  Report  (USR).  This  document,  as  prescribed  by  AR  220-1, 
provides  input  to  a  system  for  reporting  the  readiness  of  Active  and  Reserve 
Component  units.  Several  items  on  this  document  relate  directly  to  the  train¬ 
ing  readiness  of  the  reporting  unit,  including  an  estimate  of  the  number  of 
weeks  required  to  attain  a  fully  trained  status,  estimates  of  the  impacts  of 
various  constraining  factors  on  training,  and  an  overall  training  rating  on  a 
four-point  scale.  At  least  three  factors  limit  the  usefulness  of  USR  data  for 
training  feedback  purposes:  confidentiality,  generality,  and  subjectivity. 

When  USR  data  are  identified  with  a  specific  unit  and  timeframe,  they  are 
classified  Confidential.  In  order  to  avoid  working  with  classified  data,  units 
and  timeframes  must  not  be  identified,  and  this  is  often  difficult.  As  should 
be  obvious  from  the  description  above,  training  data  on  the  USR  are  very 
general,  in  terms  of  the  overall  unit  rather  than  specific  elements  or  tasks. 

The  effects  of  available  funds,  leaders,  fuel,  time,  and  similar  factors  on 
training  are  rated,  but  nothing  more  specific  is  addressed.  Most  of  the  USR 
training  data  are  provided  using  rating  scales,  so  they  represent  subjective 
estimates  on  the  part  of  the  commander,  within  the  guidelines  of  AR  220-1. 

These  guidelines  are  fairly  precise,  but  it  is  doubtful  that  USR  data  are  com¬ 
pletely  unbiased.  Consideration  of  all  the  above  factors  leads  to  the  con¬ 
clusion  that  USR  data  should  not  be  included  in  field  performance  feedback. 

The  problems  of  dealing  with  classified  data  outweigh  the  utility  of  informa¬ 
tion  available  from  the  USR. 

In  summary,  existing  unit  records  have  generally  been  found  to  be  inappro¬ 
priate  or  inadequate  as  elements  of  field  performance  feedback.  Many  of  these 
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records  are  not  task-specific,  not  always  available,  not  collected  and  main¬ 
tained  in  a  standard  form,  and  not  totally  objective  or  reliable.  Most  of 
them  are  maintained  for  reasons  other  than  feedback  to  a  Center/Scbool,  and 
such  a  use  of  them  might  lessen  their  utility  to  field  units  and  result  in 
distortion  of  data  already  having  questionable  reliability.  These  records  may 
also  not  be  timely  as  feedback  and  are  difficult  to  combine  with  other  data; 
these  issues  will  be  further  addressed  in  a  later  section  of  this  paper.  The 
only  existing  record  which  currently  appears  to  be  a  useful  feedback  element  is 
SQT  results.  Such  results  should  be  incorporated  in  an  integrated  feedback 
system,  and  they  will  be  further  discussed  in  late'*  sections  on  testing  and 
overall  system  design.  ARTEP  results  also  offer  some  promise  as  feedback  ele¬ 
ments  in  the  future,  once  collective  and  individual  tasks  are  clearly  inter¬ 
related.  The  derivation  of  individual  task  results  from  collective  exercises 
will  also  be  further  discussed  later  in  this  paper.  But  aside  from  SQT  and 
perhaps  ARTEP  results,  existing  records  are  not  useful  elements  of  field  per¬ 
formance  feedback. 

Surveys/Questionnaires.  Probably  the  most  popular  method  of  gathering 
field  performance  feedback  in  the  past  has  been  through  the  administration  of 
surveys  to  field  personnel.  Interviews  with  battalion  commanders  revealed 
that  this  is  the  predominant  method  used  by  Centers/Schools  to  solicit  feed¬ 
back.  During  the  past  year,  five  of  seven  commanders  had  completed  a  survey 
on  the  quality  of  graduates  and  three  had  completed  one  on  .the  usefulness  of 
training  materials.  In  addition,  personnel  throughout  battalions  frequently 
respond  to  surveys  on  the  quality  of  training  they  have  received  in  various 
courses  or  on  the  training  received  by  personnel  they  supervise.  It  appears 
that  almost  every  Center/School  in  the  Army  sends  surveys  to  the  field,  and 
that  field  personnel  are  sometimes  swamped  by  them.  It  is  interesting  to  note 
that  TRAuOC  Regulation  350-6  does  not  require  surveys  or  prescribe  procedures 
for  administering  them,  although  draft  versions  of  this  regulation  did.  Since 
feedback  surveys  are  so  widespread,  appropriate  regulatory  guidance  for  their 
use  should  be  developed.  In  the  case  of  Armor,  extensive  surveys  are  utilized 
by  OAFMS  in  conjunction  with  interviews  to  gather  feedback  from  the  field.  In 
ARTS,  LST,  and  other  research  projects,  general  surveys  have  typically  been 
used  as  a  method  of  assessing  the  state  of  training  in  field  units.  Surveys 
are  also  widely  used  in  services  other  than  the  Army;  for  example.  Dyer,  Mew, 
and  Ryan  (1975)  have  developed  an  extensive  procedures  manual  for  utilization 
of  questionnaires  in  training  feedback  within  the  US  Navy.  But  the  wide  use 
of  surveys  does  not  signify  that  they  are  the  perfect  solution  to  the  problem 
of  obtaining  training  feedback.  There  are  many  positive  and  negative  aspects 
of  survey  usage  which  should  be  considered. 

With  respect  to  the  format  of  surveys,  a  primary  issue  is  whether  or  not 
they  are  task-specific.  Responses  are  typically  obtained  in  terms  of  five  or 
six-point  rating  scales,  and  questions  can  vary  in  scope  from  very  broad  to 
very  specific.  In  research  such  as  ARTS  and  the  application  of  LST,  questions 
have  typically  been  very  broad,  such  as  "What  is  the  present  state  of  training 
in  your  battalion?"  or  "How  clear  are  your  training  missions?"  Questions  on 
surveys  from  Centers/ Schools  are  sometimes  also  very  broad,  such  as  "How  well 
trained  are  personnel  when  they  arrive  at  your  unit?"  On  the  other  hand, 
survey  questions  can  be  very  specific.  For  example,  questionnaires  used  by 
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OAFMS  and  the  Navy  (Dyer,  Mew,  and  Ryan,  1975)  ask  respondents  to  rate  the 
criticality,  frequency,  and  performance  of  each  task  covered  in  the  training 
base.  Answers  are  given  with  respect  to  a  specific  graduate  or  the  typical 
graduate;  in  either  case,  detailed  task-specific  data  can  be  gathered  via 
survey  techniques.  The  question  which  thus  arises  concerns  whether  questions 
on  feedback  surveys  should  be  general  or  task-specific  in  nature.  Obviously, 
responses  to  task-specific  questions  are  much  more  valuable  as  feedback,  since 
they  indicate  much  more  precisely  where  problems  lie  that  need  to  be  addressed. 
Responses  to  general  survey  questions  are  often  not  very  useful,  since  they  do 
not  indicate  specific  actions  to  be  taken.  But  task-specific  surveys  often 
run  the  risk  of  being  too  lengthy  and  cumbersome.  For  example,  some  question¬ 
naires  utilized  by  OAFMS  include  over  100  tasks  and  require  over  300  responses. 
This  may  be  too  many  tasks  for  anyone  in  the  field  to  observe  and  evaluate, 
and  completing  this  long  a  questionnaire  may  be  so  exhausting  as  to  influence 
the  quality  of  responding.  The  solution  to  this  dilemma  would  seem  to  be  the 
reaching  of  a  happy  medium.  Survey  questions  should  be  as  task-specific  as 
possible,  but  not  all  the  tasks  within  a  particular  duty  may  need  to  be  evalu¬ 
ated.  Respondents  could  be  presented  with  subsets  of  tasks  to  evaluate,  or 
only  tasks  vhich  have  given  some  other  indication  of  being  a  problem  could  be 
addressed.  That  is,  tasks  could  be  addressed  by  exception  rather  than  in  toto. 
Also,  the  rating  scales  used  could  probably  be  simplified  to  three-point 
scales  without  any  loss  of  information,  and  issues  such  as  frequency  and  criti¬ 
cality  of  task  performance  would  not  have  to  be  rated  at  the  same  time  as 
quality  of  performance.  In  conclusion,  feedback  surveys  should  be  task-specific 
as  long  as  they  do  not  become  too  cumbersome.  This  point  will  be  considered  in 
feedback  system  design  later  in  this  paper. 

The  major  advantage  of  using  surveys  in  field  performance  feedback  is 
that  they  are  relatively  fast  and  inexpensive.  Even  lengthy  task-specific 
surveys  can  be  completed  in  an  hour  or  two,  and  the  use  of  mailed  surveys 
eliminates  travel  costs  and  minimizes  resource  expenditures.  But  there  are 
several  negative  aspects  to  the  use  of  surveys.  Perhaps  foremost  among  these 
is  that  the  data  gathered  in  surveys  are  subjective  perceptions  which  may  not 
reflect  reality.  Respondents  are  usually  asked  to  evaluate  how  well  they 
themselves  or  people  working  for  them  can  perform  certain  tasks,  and  such  esti¬ 
mates  are  by  no  means  perfect,  especially  if  a  large  number  of  infrequently 
performed  tasks  is  being  addressed.  However,  some  senior  military  personnel 
will  argue  that  perceptual  data  provide  as  valid  an  assessment  of  training  ef¬ 
fectiveness  as  any  other  available  measure  does.  Whether  this  augurs  well  for 
perceptual  data  or  poorly  for  other  measures  is  a  debatable  point.  The  accu¬ 
racy  of  survey  estimates  could  be  ascertained  by  administering  hands-on  tests 
for  a  sample  of  the  tasks  addressed;  i.e.,  a  combination  of  the  survey  approach 
and  the  testing  approach  to  be  discussed  below  might  be  an  appropriate  feedback 
method.  Written  (paper-and-pencil)  tests  could  be  used  in  lieu  of  hands-on 
tests  for  some  tasks,  and  the  relationship  of  survey  estimates,  written  test 
performance,  and  hands-on  performance  could  thus  be  determined  for  various 
tasks.  Hall,  Denton,  and  Zajkowski  (1978)  indirectly  compared  proficiency 
estimates  from  surveys  and  results  of  written  tests  and  found  no  correlation 
between  them,  indicating  that  a  further  look  should  be  taken  at  the  accuracy 
of  survey  responses.  More  on  this  later,  but  for  present  purposes  the  impor¬ 
tant  point  is  that  survey  responses  should  not  be  used  as  feedback  without 
some  means  of  checking  their  accuracy.’ 
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Other  potential  problems  with  the  survey  approach  to  feedback  will  only 
briefly  be  covered  here.  One  of  these  is  determining  the  appropriate  sample 
to  be  surveyed.  The  sample  chosen  should  be  a  representative  one  and  should 
not  include  only  the  senior  personnel  in  a  unit,  since  they  may  not  have  an 
opportunity  to  observe  task  performance  on  a  regular  basis.  Related  to  this 
is  the  problem  of  return  rate.  Dyer,  Ryan,  and  Mew  (1975)  found  that  super¬ 
visors  return  mailed  surveys  at  a  rate  of  60%,  while  trainees  return  only  30%. 
OAFKS  has  found  similar  results  and  thus  administers  surveys  to  lower-ranking 
personnel  during  visits  to  units  and  not  through  the  mail.  Hall,  Denton,  and 
Zajkowski  (1978)  found  that  satisfactory  survey  return  rates  could  only  be  ob¬ 
tained  by  administering  the  surveys  during  visits  to  the  field.  Surveys  can 
also  be  administered  to  senior  NCO's  and  officers  while  they  are  attending 
courses  at  the  Center/School,  thus  saving  mail  and/or  travel  costs.  Of  course, 
the  data  obtained  by  this  method  may  not  be  as  accurate  as  those  gathered  in 
the  field,  since  they  are  based  upon  longer-term  memory.  Decisions  must  be 
made  in  all  these  areas  depending  upon  the  resources  available  and  the  return 
rate  and  amount  of  detail  in  responses  needed.  A  final  common  problem  with 
survey  data  is  that  averages  of  rating  scale  responses  tend  to  cluster  around 
the  mid-point  of  the  scale.  Differences  in  average  responses  between  units 
may  thus  be  small,  but  they  can  be  statistically  significant  if  enough  data 
arc  collected.  One  must  consider  the  issue  of  practical  significance  in  the 
design  of  a  survey  and  selection  of  a  sample,  so  that  statistically  significant 
differences  are  practically  meaningful. 

In  conclusion,  surveys  have  been  a  major  part  of  field  performance  feed— 
rir.ck  and  will  continue  to  be  so,  because  of  their  low  cost.  But  they  are  not 
^Icimate  solution  to  the  feedback  problem,  due  to  the  numerous  problems 
inherent  in  survey  administration  (as  suggested  earlier,  these  problems  could 
be  reduced  by  provision  of  appropriate  regulatory  guidance  for  survey  adminis¬ 
tration).  Surveys  should  be  used  as  only  one  part  of  an  integrated  feedback 
systv^n.  Perhaps  limited  task-specific  surveys  could  be  used  to  provide  initial 
indications  of  problems  to  be  further  addressed  by  approaches  described  below. 

Interviews.  The  conduct  of  interviews  to  gather  feedback  information  is 
an  approach  closely  related  to  administration  of  surveys,  but  not  as  widely 
used.  The  primary  differences  are  that  an  interview  is  generally  conducted  in 
a  face-to-face  setting  and  requires  verbal  rather  than  written  responses.  The 
relatively  low  use  of  interviews  probably  relates  tc  the  fact  that  they  are 
more  resource- intensive  than  are  surveys.  Interviews  with  battalion  commanders 
indicated  that  Centers/Schools  do  not  extensively  use  interview  methods  to 
gather  feedback  from  the  field;  an  average  of  less  than  one  interview  was  con¬ 
ducted  per  battalion  in  the  past  year.  Interviews  are  utilized  to  gather  feed¬ 
back  by  OAFMS  personnel;  they  interview  approximately  10  personnel  in  each 
Armor  battalion  visited.  The  Navy  also  uses  this  method  to  some  extent;  Hall 
and  Hughes  (1980)  have  developed  an  approach  for  interviewing  petty  officers 
attending  advanced  schools.  It  is  interesting  to  note  that  almost  all  of  the 
seven  battalion  commanders  interviewed  to  provide  input  for  this  paper  stated 
that  they  prefer  to  participate  in  an  interview  rather  than  a  survey.  Field 
commanders  complete  many  questionnaires  in  today's  Army,  and  they  admit  that 
they  sometimes  become  fatigued  by  them  and  do  not  devote  as  much  thought  to  " 
then  as  they  could.  But  the  general  response  to  an  interview  is:  "If  you're 
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going  to  take  the  time  to  talk  to  me,  then  I'll  cooperate  fully."  Respondents 
are  thus  likely  to  provide  more  detailed  comments  in  an  interview  than  on  a 
questionnaire.  Also,  many  personnel  in  today's  Army  have  limited  writing 
skills,  especially  among  the  lower  enlisted  ranks.  The  oral  interview  is  thus 
definitely  a  better  method  than  the  written  survey  for  collection  of  data  from 
such  personnel.  The  interview  is  certainly  an  appropriate  feedback  collection 
method,  and  it  should  perhaps  be  used  more  widely,  if  resources  allow. 

Primary  issues  relating  to  the  conduct  of  interviews  include  the  degree  of 
structure,  the  amount  of  detail,  and  the  site  of  the  interview.  Interviews  are 
generally  categorized  as  structured  or  unstructured,  with  structured  ones  fol¬ 
lowing  a  fairly  rigid  format  and  unstructured  ones  being  more  of  a  free-flowing 
conversation.  The  structured  interview  approach  is  more  appropriate  here, 
since  feedback  is  generally  sought  on  specific  issues  and  more  specific  in¬ 
formation  can  be  obtained  by  using  a  structured  format.  However,  the  structure 
should  not  be  so  rigid  as  to  stifle  voluntary  comments.  Interviews  should  also 
be  as  task-specific  as  possible,  since  comments  will  be  more  useful  if  they 
relace  to  specific  tasks.  To  accomplish  this,  interviews  can  be  conducted 
following  surveys  or  some  other  indication  of  tasks  which  may  be  key  problem 
areas.  Like  surveys,  interviews  can  be  conducted  either  in  field  units  or 
while  personnel  are  attending  advanced  courses  at  Centers/Schools.  The  latter 
approacn  has  the  advantage  of  minimizing  travel  costs,  but  the  disadvantage  of 
gathering  responses  based  upon  relatively  long-term  memory.  An  interview  ap¬ 
proach  must  include  methodology  for  gathering  detailed  responses  from  personnel 
while  they  are  in  field  units,  to  the  extent  that  resources  allow.  A  final 
concern  previously  discussed  with  respect  to  surveys  should  also  be  mentioned 
here,  and  this  is  the  accuracy  or  validity  of  interview  data.  Hall,  Denton, 
and  Zajkowski  (1978)  found  no  correlation  between  proficiency  estimates  ob¬ 
tained  during  interviews  and  performance  on  written  tests.  Their  interview 
was  essentially  an  oral  questionnaire,  and  it  would  be  more  conclusive  to  com¬ 
pare  interview  estimates  with  hands-on  rather  than  written  tests,  but  their 
results  indicate  that  the  accuracy  of  interview  data  should  be  further  examined. 

In  summary,  the  interview  approach  should  be  included  in  a  field  perform¬ 
ance  feedback  system,  since  it  may  allow  collection  of  more  detailed  data  than 
the  survey  approach  and  it  is  more  acceptable  to  many  field  personnel.  Inter¬ 
views  should  be  structured,  task-specific,  and  conducted  in  the  field  to  the 
extent  possible.  Since  interviews  are  relatively  resource- intensive,  they 
s.iouid  be  used  in  conjunction  with  other  methods  to  ensure  that  interviews  are 
directed  at  key  problem  areas  and  to  check  the  accuracy  of  interview  data. 

Observation  of  Performance.  If  a  Center/School  was  concerned  about  the 
proficiency  of  its  graduates  or  the  usefulness  of  its  training  materials  in 
the  field,  a  logical  approach  to  addressing  this  concern  would  be  to  send  per¬ 
sonnel  to  field  units  to  observe  performance.  Indications  are  that  this  does 
not  occur  frequently.  The  seven  battalion  commanders  interviewed  reported  no 
visits  by  Center/School  personnel  to  observe  performance  during  the  past  year. 
They  also  reported  that  they  had  not  been  required  to  observe  performance  and 
develop  data  for  a  Center/School.  The  limited  use  of  this  methodology  is 
prob^bay  due  to  resource  and  scheduling  constraints.  Center/School  personnel 
generally  do  not  have  the  time  or  travel  funds  available  for  the  length  of  tine 
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required  to  observe  and  record  performance.  Training  schedules  of  field  units 
are  usually  very  dynamic  and  it  would  be  difficult  to  coordinate  a  schedule  of 
visits  to  a  number  of  units  undergoing  a  similar  major  training  event.  In 
addition,  data  collection  visits  are  often  perceived  as  training  dirtractors 
by  field  personnel,  so  the  potential  benefits  of  such  visits  must  be  clearly 
explained.  But  the  only  way  to  know  for  sure  what  is  happening  in  the  field 
is  to  go  out  and  see,  so  methodology  for  observation  of  performance  s.  ould  be 
included  in  a  field  performance  feedback  system. 

A  major  issue  relating  to  this  approach  is  how  to  derive  individual  task 
data  from  observation  of  collective  exercises.  The  resources  available  to 
Centers/Schools  do  not  allow  extensive  observation  of  individual  task  perform- 
ance  in  the  field,  but  the  observation  of  collective  exercises  could  poten¬ 
tially  provide  a  large  amount  of  data  during  a  relatively  short  period  of  time. 
Surveys  and  interviews  could  also  be  conducted  immediately  after  such  exercises 
to  collect  data  on  specific  problems  observed  and  to  reduce  memory  problems 
mentioned  in  the  previous  discussions  of  other  methodologies.  For  example,  in 
Armor  battalions  the  performance  of  crew  drills  and  platoon  battle  runs  could 
be  observed  and  follow-up  instruments  could  be  tailored  to  these  exercises. 
Objective  criteria  for  observation  of  such  performance  need  to  be  developed, 
along  with  methods  for  relating  collective  performance  to  the  accomplishment 
or  failure  of  individual  tasks.  As  mentioned  earlier,  the  relationships  of 
ARTEP  and  individual  tasks  are  becoming  more  clearly  defined  (for  example,  the 
Infantry  School  has  developed  a  Student  Text  (ST  21-6-188-1)  detailing  such 
relationships),  and  work  is  also  progressing  on  further  specification  of  crew 
drills  and  platoon  exercises.  A  field  performance  feedback  system  should  be 
integrated  with  such  exercises  in  order  to  optimize  the  utility  and  cost- 
effectiveness  of  field  observation. 

Field  Testing.  The  ultimate  method  for  Centers/Schools  to  determine  the 
proficiency  of  their  graduates  would  be  to  follow  them  to  the  field  and  test 
them.  Such  tests  would  ideally  be  unannounced,'  hands-on,  task-specific,  and 
conducted  by  impartial  testers.  That  is,  soldiers  would  be  given  a  "pop  quiz." 
Interviews  with  battalion  commanders  and  previous  research  experience  indicate 
that  this  approach  to  gathering  feedback  is  rarely  utilized.  The  closest 
method  in  use  is  the  administration  of  SQT*s,  which  were  previously  discussed 
as  a  type  of  existing  record.  SQT’s  are  not  unannounced  and  are  not  entirely 
hands-on,  but  these  may  not  be  critical  problems.  Conducting  unannounced  "pop" 
tests  would  be  practically  impossible  and  would  be  disruptive  of  the  unit 
schedule;  all  that  is  required  is  that  soldiers  not  know  exactly  what  is  to  be 
tested  and  not  be  trained  for  just  those  tasks.  As  discussed  earlier,  written 
tests  are  an  acceptable  ?nproach,  as  long  as  they  are  validated  against  opera¬ 
tional  performance.  SQT's  may  thus  provide  an  acceptable  approach  to  gathering 
l<_  l  data  as  field  performance  feedback,  if  tasks  to  be  tested  arc  not  specif¬ 
ically  announced  and  prepared  for. 

However,  in  some  cases  it  might  prove  advantageous  for  Centers/Schools  to 
collect  test  data  in  addition  to  SQT  results.  Additional,. more  detailed  data 
might  be  needed  on  specific  tasks  which  surface  as  problem  areas,  or  data 
might  be  desired  for  checking  the  accuracy  or  reliability  of  SQT  results. 

Such  data  might  also  be  used  to  validate  subjective  estimates  obtained  from 
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surveys  or  interviews.  For  example,  survey  estimates  of  soldiers*  ability  to 
perform  specific  tasks  could  be  compared  with  results  of  performance  tests. 
Whatever  the  reason  for  its  collection,  additional  performance  test  data  cannot 
be  extensively  obtained  by  Centers/Schools,  because  of  the  resource  require¬ 
ments  involved.  Previous  research,  such  as  ARTS,  has  shown  that  testers' 
biases  can  only  be  controlled  by  using  a  constant  team  of  testers  to  collect 
performance  data.  Moving  a  team  of  testers  from  a  Center/School  to  various 
installations  is  not  practical.  In  addition  to  the  need  for  testers,  the  col¬ 
lection  of  hands-on  performance  data  will  tie  up  a  good  deal  of  a  unit's  time 
and  equipment.  This  approach  thus  does  not  meet  two  of  the  criteria  mentioned 
in  the  introductory  section  of  this  paper;  i.e.,  it  may  not  be  doable  within 
current  resources  and  it  could  be  viewed  as  a  detractor  to  unit  training. 
Eauds-on  performance  test  data  in  addition  to  SQT  results  should  thus  only  be 
collected  in  special  situations  where  training  problems  have  previously  been 
identified,  or  in  research  on  comparison  of  different  types  of  performance 
measures.  Such  data  could  also  perhaps  be  collected  by  administering  tests  to 
students  in  process  for  professional  development  courses.  Hall,  Denton,  and 
Zajkowski  (1978)  provide  data  demonstrating  the  high  cost  of  testing  as  com¬ 
pared  to  survey  and  interview  methods,  and  they  agree  with  the  conclusion 
reached  here  by  recommending  that  the  testing  approach  to  feedback  be  used 
only  in  limited  situations. 

Note  that  the  testing  approach  to  gathering  feedback  data  applies  only  to 
assessing  the  proficiency  of  graduates,  and  not  to  assessing  the  utility 
training  materials.  The  latter  assessment  will  have  to  be  accomplished  through 
observation,  survey,  and  interview  techniques.  Conducting  detailed  observa¬ 
tions  of  normal  unit  exercises  and  testing  will  allow  Oenters/Schools  to  evalu¬ 
ate  the  usefulness  of  materials  and  will  probably  minimize  the  need  for  addi¬ 
tional  testing  of  personnel.  In  the  feedback  system  of  the  future,  performance 
testing  should  be  reserved  as  a  last  option  to  obtain  feedback  in  specific 
problem  areas. 

MAJOR  FEEDBACK  ISSUES 

In  the  previous  section  six  methods  of  collecting  field  performance  feed¬ 
back  were  reviewed  and  all  were  found  to  be  useful,  at  least  to  a  limited  ex¬ 
tent.  In  the  next  section  initial  proposals  will  be  developed  for  improving 
feedback  and  developing  an  integrated  system  combining  these  methods.  But  be¬ 
fore  doing  this,  several  major  issues  and  their  impact  on  a  feedback  system 
should  be  discussed.  Many  of  these  issues  were  touched  upon  in  the  previous 
section,  but  they  need  to  be  further  considered  in  general  rather  than  in  the 
context  of  a  specific  feedback  approach.  These  issues  are  briefly  discussed 
below. 

One  major  issue  relating  to  field  performance  feedback  is  the  question  of 
to  whom  it  should  be  provided  or,  alternatively,  what  schools  should  be  in¬ 
cluded  in  a  feedback  system.  The  answer  is  basically  all  of  them.  As  men¬ 
tioned  in  the  introductory  section,  a  general  external  feedback  approach  should 
apply  to  all  levels  of  institutional  training,  from  initial  entry  training  to 
the  US  Army  War  College.  It  should  also  apply  to  all  military  specialties, 
both  officer  and  enlisted.  The  application  of  feedback  methods  reviewed  in 
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this  paper  has  thusfar  been  concentrated  upon  institutional  training  of  junior 
enlisted  personnel  and,  in  some  cases,  junior  officers.  Some  of  the  commanders 
interviewed  to  provide  background  for  this  paper  expressed  the  view  that  feed¬ 
back  to  Centers/Schools  should  be  limited  to  basic  entry  skills,  with  higher 
level  skills  being  the  responsibility  of  the  unit.  But  this  is  not  the  case; 
rather,  feedback  should  be  provided  to  all  levels  of  schools,  including  those 
providing  basic  and  advanced  NCO  training  and  advanced  officer  training.  The 
tasks  to  be  addressed  in  feedback  are  thus  not  just  those  at  the  basic  entry 
level,  but  all  types  of  tasks,  including  command,  control,  and  communications. 
This  leads  to  the  need  to  integrate  individual  and  collective  tasks  in  a  feed¬ 
back  system,  which  will  be  discussed  as  a  separate  issue  below.  The  next 
question  relates  to  who  within  a  Center/School  should  be  provided  feedback. 

The  optimal  solution  appears  to  be  the  establishment  of  a  separate  element  for 
t'his  purpose,  such  as  OAFMS.  In  this  way  appropriate  resources  can  be  devoted 
to  gathering  and  collating  feedback  data  on  a  long-term  basis.  This  office 
should  also  have  the  authority  to  follow-up  changes  recommended  based  upon  the 
data  gathered  and  to  ensure  that  these  changes  are  made. 

As  mentioned  earlier,  another  major  issue  is  the  gathering  and  integration 
of  both  collective  and  individual  task  data.  Collective  data  are  important  to 
collect  in  their  own  right,  since  they  are  important  feedback  to  advanced 
courses  of  instruction  and  to  developers  of  collective  training  materials. 
Gathering  data  during  or  shortly  after  collective  exercises  may  also  provide  a 
more  economical  means  of  collecting  individual  task  data,  if  methods  can  be 
developed  for  deriving  individual  task  performance  from  collective  results. 
Progress  has  been  made  in  this  direction,  as  exemplified  by  the  Infantry 
School’s  ST  21-6-188-1,  so  future  feedback  research  will  address  collection  of 
both  individual  and  collective  data.  In  the  past,  feedback  approaches  have 
concentrated  too  heavily  upon  individual  task  data. 

Another  issue  of  concern  is  the  timing  of  feedback  data  collection.  One 
question  which  has  frequently  arisen  in  the  past  concerns  when  feedback  should 
be  gathered  on  the  proficiency  of  Center/School  graduates.  The  answer  has 
generally  b^en  three  to  six  months  after  graduation,  so  that  ample  time  has 
been  available  to  perform  in  t^e  unit.  This  is  probably  an  appropriate  ap¬ 
proach,  although  in  some  cases  longer  term  data  collection  might  be  desirable. 
But  this  solution  applies  basically  to  only  survey  methodology.  When  should 
an  integrated  collection  of  individual  and  collective  task  data  of  various 
types  take  place?  The  answer  seems  to  be  during  and  after  major  collective 
exercises.  Observations  and  unsolicited  comments  could  be  gathered  during  such 
exercises,  and  existing  records,  questionnaire  responses,  and  interview  com¬ 
ments  could  be  gathered  shortly  afterwards.  In  this  way,  records  which  are 
highly  perishable  could  be  obtained  and  memory  failures  of  respondents  could 
be  minimized.  Much  work  will  be  required  to  determine  optimal  schedules  for 
integrated  feedback  data  collection,  but  such  collection  will  benefit  from  be¬ 
ing  coordinated  with  unit  activities  to  the  maximum  extent  possible. 

If  one  has  gathered  a  large  amount  of  various  types  of  feedback  data,  how 
can  it  be  collated,  analyzed,  or  otherwise  combined  to  support  appropriate  de¬ 
cisions?  OAFMS  has  collected  a  large  amount  of  survey  and  interview  data,  but 
has  not  yet  found  a  way  to  combine  and  maintain  them  to  establish  long-term 


16 


■ it*  * 


trends.  LST  research  has  shown  that  survey  data  and  data  from  existing  records 
do  not  always  correlate  highly,  and  O'Mara  (1981)  has  shown  that  elements  of 
existing  records  do  not  have  high  intercorrelations.  The  problem  is  thus  to 
form  composites  of  large  amounts  of  data  which  may  be  disparate.  No  solution 
to  this  issue  can  be  offered  at  present,  except  to  point  out  that  computer 
support  will  be  needed  and  to  suggest  that  data  be  organized  by  specific  tasks. 
The  data  management  system  should  also  include  a  method  to  update  the  data 
base  by  replacing  the  oldest  data  with  new.  That  is,  data  over  a  year  old 
might  be  purged  from  the  system  as  new  data  are  added.  Data  management  will 
be  a  primary  focus  of  future  research  on  field  performance  feedback,  especially 
with  regard  to  comparison  and  combination  of  different  types  of  data. 

Throughout  this  paper  the  collection  of  feedback  data  on  the  proficiency 
of  Center/School  graduates  and  the  usefulness  of  training  materials  have  been 
discussed  largely  in  conjunction.  In  reality,  somewhat  different  approaches 
must  be  developed  for  gathering  feedback  in  these  two  areas.  Interviews  of 
battalion  commanders  and  a  review  of  the  literature  indicate  that  feedback 
systems  have  in  the  past  concentrated  upon  the  collection  of  data  relating  to 
the  evaluation  of  graduates.  Feedback  in  the  future  should  more  formally  ad¬ 
dress  the  evaluation  of  training  materials,  through  the  approaches  of  surveys, 
structured  interviews,  and  observation  of  field  performance.  The  observational 
approach  appears  to  be  an  especially  fruitful  one  which  should  be  used  more 
widely  here.  Materials  cannot  be  tested  in  the  same  way  that  personnel  can, 
but  their  use  can  be  monitored  and  objectively  evaluated.  The  gathering  of 
feedback  data  during  and  immediately  after  the  use  of  training  materials  in 
field  exercises  may  provide  a  context  for  addressing  both  areas  of  concern. 

Another  major  issue  is  assuring  that  feedback  has  the  proper  impact. 

Several  measurement  and  decision-making  issues  which  have  been  mentioned  pre¬ 
viously  are  relevant  here,  and  they  are  briefly  summarized  below.  Feedback 
must  validly  and  reliably  represent  the  situation  in  the  field,  and  it  should 
be  as  objective  as  possible,  in  order  to  allow  combination  of  data  and  drawing 
of  proper  conclusions.  A  feedback  system  should  thus  not  rely  entirely  upon 
the  gathering  of  subjective  survey  and  interview  data,  since  research  by  Kali, 
Denton,  and  Zajkowski  (1978)  raises  doubts  about  the  accuracy  of  such  responses. 
Feedback  must  be  task-specific  in  nature,  so  that  precise  follow-up  actions 
can  be  taken.  It  must  be  based  upon  a  large  enough  sample  to  represent  large- 
scale  problems  in  the  field,  and  not  just  problems  in  selected  units.  Appro¬ 
priate  sampling  criteria  must  be  developed  and  followed,  so  that  premature 
decisions  are  not  reached,  based  upon  preliminary  data.  Criteria  must  also  be 
developed  for  reaching  sound  decisions  once  sufficient  samples  of  data  are 
gathered.  And  finally,  actions  and  revisions  developed  must  be  doable  within 
available  resources  and  must  be  monitored  to  ensure  proper  implementation. 

An  issue  closely  related  to  the  previous  one  of  assuring  that  feedback 
has  the  proper  impact  is  organizational  capability  to  manage  a  feedback  system. 
The  typical  Army  Center/School  does  not  have  the  personnel  or  skills  available 
to  adequately  conduct  feedback  planning,  collection,  analysis,  and  interpreta¬ 
tion.  If  the  Army  wants  a  useful  feedback  system,  adequate  resources  must  be 
devoted  to  it  at  each  Center/School.  These  resources  should  include  a  team  of 
well-trained  data  collectors  to  gather  data  from  the  field,  a  data  analyst  with 
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automated  data  processing  experience,  and  a  system  manager  who  can  properly 
interpret  findings  and  communicate  them  to  nontechnical  personnel.  These  re¬ 
sources  should  be  civilian  or  stabilized  military  personnel  who  will  not  be 
reassigned  just  as  they  are  developing  useful  feedback  skills;  a  mix  of  mili¬ 
tary  and  civilian  personnel  would  probably  be  ideal.  Resource  requirements 
will  vary  somewhat  in  different  Centers/Schools,  and  they  will  be  addressed  in 
further  detail  during  the  research  described  below. 

The  final  issue,  which  has  been  a  predominant  one  throughout  this  paper, 
is  the  integration  of  various  approaches  to  the  collection  of  field  performance 
feedback.  Six  approaches  have  been  identified  which  have  utility  in  certain 
situations.  They  can  be  integrated  by  tailoring  them  both  to  the  resources 
available  and  the  severity  of  the  problem  identified.  That  is,  problems  can 
initially  be  identified  by  review  of  unsolicited  comments,  existing  records, 
and  survey  responses.  Serious  problems  can  then  be  further  addressed  through 
structured  interviews,  observation,  and,  if  necessary,  field  testing.  Such  an 
overall  approach  will  be  a  focus  of  further  research  in  the  area  of  feedback, 
as  discussed  in  the  following  section. 

FUTURE  RESEARCH  DIRECTIONS 

In  order  to  support  development  of  an  integrated  approach  to  field  per¬ 
formance  feedback,  research  following  up  on  this  paper  will  be  concentrated  in 
at  least  three  directions:  the  collection  of  feedback  during  and  after  field 
exercises,  the  comparison  of  data  gathered  from  different  feedback  approaches, 
and  the  management  of  feedback  data.  General  plans  for  research  in  each  of 
these  directions  are  briefly  described  below. 

An  ihitial  effort  to  collect  feedback  data  from  collective  exercises  will 
be  conducted  within  the  context  of  crew  and  platoon  level  exercises  in  Armor 
battalions.  Based  upon  documentation  that  presently  exists  or  is  being  de¬ 
veloped,  the  individual  tasks  involved  in  crew  drills  and  platoon  battle  runs 
will  be  delineated.  Methods  for  observing  and  recording  performance  on  these 
tasks  in  as  objective  a  manner  as  possible  will  be  developed  and  applied  during 
appropriate  exercises.  Structured  interviews  will  also  be  developed  and  will 
be  conducted  with  participants  shortly  after  exercise  completion.  These  inter¬ 
views  will  be  directed  toward  particular  problems  that  occurred  during  the 
preceding  exercise,  rather  than  toward  an  exhaustive  review  of  all  tasks  that 
were  performed.  In  this  way,  the  interview  approach  will  be  made  wore  efficient 
and  less  dependent  upon  long-term  memory.  Unsolicited  comments  and  existing 
records  will  also  be  gathered  during  the  crew  and  platoon  exercises  as  they 
become  available,  in  order  to  determine  their  utility  as  feedback  and  to  corn- 
par,:  them  to  interview  and  observational  results.  As  a  result  of  this  entire 
effort,  feasible  methods  of  gathering  individual  and  collective  performance 
feedback  from  crew  and  platoon  level  exercises  will  be  tested,  compared,  and 
refined. 

The  comparison  of  data  collected  via  different  feedback  approaches  will 
address,  as  a  minimum,  the  results  of  surveys,  interviews,  and  hands-on  tests. 
Personnel  in  Armor  battalions  will  be  asked,  through  a  survey  or  interview,  to 
estimate  the  proficiency  with  which  they  or  personnel  working  for  them  can 
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automated  data  processing  experience,  and  a  system  manager  who  can  properly 
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overall  approach  will  be  a  focus  of  further  research  in  the  area  of  feedback, 
as  discussed  in  the  following  section. 
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formance  feedback,  research  following  up  on  this  paper  will  be  concentrated  in 
at  least  three  directions:  the  collection  of  feedback  during  and  after  field 
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be  conducted  with  participants  shortly  after  exercise  completion.  These  inter¬ 
views  will  be  directed  toward  particular  problems  that  occurred  during  the 
preceding  exercise,  rather  than  toward  an  exhaustive  review  of  all  tasks  that 
were  performed.  In  this  way,  the  interview  approach  will  be  made  wore  efficient 
and  less  dependent  upon  long-term  memory.  Unsolicited  comments  and  existing 
records  will  also  be  gathered  during  the  crew  and  platoon  exercises  as  they 
become  available,  in  order  to  determine  their  utility  as  feedback  and  to  com¬ 
pare  them  to  interview  and  observational  results.  As  a  result  of  this  entire 
effort,  feasible  methods  of  gathering  individual  and  collective  performance 
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